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ABSTRACT 

Throwing the proverbial monkey vrench into the 
machinery **on paper^ before a system begins functioning seems to be 
essential to program evaluation. This activity should be performed by 
the ^resident skeptic" who is familiar with past resistance to change 
and whOt being familiar with a program, is able to foresee 
theoretical and practical problems* Such creative pessimism should be 
applied to the evaluation of Title One Programs in a large urban 
district* It may prove useful to categorize anticipated concerns and 
potential problems under six general headings: (1) Planning: includes 
providing useful data within a mandated time frame and specification 
by the Federal Government of minimum evaluation requirements # (2) 
Organization: includes setting up an organizational linkage between 
the evaluation and instructional arms of a school system, (3) 
Implementation: includes considering evaluation results in program 
and management decisions and combating outside criticise that 
research and evaluation activities constitute a frill, (4) 
Bissemination: includes avoiding gross oversimplification of research 
results, (5) Personnel: includes convincing personnel that program 
evaluation should not be equated with personnel appraisal and (6) 
Siscellaneous: includes addressing demands for non-existent 
information, adjusting to the Buckley amendment and contracting r/ith 
outside firms for evaluation services* (Author/BJ6) 
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APPROACHING TITLE ONE PROGRAM EVALUATION 
WITH CREATIVE PESSIMISM 

by 

Michael H. Kean 

Whoever Murphy was, he could not have envisioned that the humorous 
law which bears his name would be so appropos during this age of tech- 
nology and complex organizations. Murphy's law, of course, states that 
in every given situation, anything that possibly can go wrong, will. 
The education of disadvantaged children is far too important to leave 
to chance, however. So, armed with the wisdom of Murphy's law, it is 
incumbent upon us to prepare for potential problems, not wait to be 
battered by them. 

It is my hypothesis that an approach which I shall call "creative 
pessimism" may serve as a vehicle to identify such areas of concern. 
Creative pessimism is the process of deliberately establishing a series 
of potential obstacles of sufficient magnitude, so that if not removed 
they would prevent anticipated events from occurring. More simply 
stated, creative pessimism is the act of purposely throwing the proverbial 
monkey wrench into the machinery, but "on paper;" not once the system 
is already functioning. 

In practicing the act of creative pessimism, one might function in 
the role of "resident skeptic." Such an individual serves the purpose 
of communicating the fruits of creative pessimism to those empowered to 
take action. The resident skeptic must be sufficiently familiar with a 
project or program to identify problems of the highest order — both 
theoretical and practical. He should not, however, be convinced of its 
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merit, but Instead should stand on the negative side of the neutral 
line. Another way to view the resident skeptic, particularly when dealing 
with Title One Programs, is as a modified ombudsman — a person who points 
out any and all potential problems, no matter what or whom they might 
impact upon, and irregardless of which group or individual would assume 
responsibility for remedying them. 

A prudent manager, charged with the responsibility of implementing 
a new program, particularly if it is innovative in nature, should review 
thoroughly the literature and research dealing with resistance to change, 
so that he will be in a position to minimize its deleterious effects. 
This exemplifies the proactive decision mode as opposed to the reactive 
mode, and carries with it all of the advantages which does planning 
versus non-planning. Creative pessimism provides another way of 
approaching resistance to change, as long as it is practiced by design, 
with the necessary linkages having been attended to In advance. 
Spontaneous, uncontrolled pessimism can quickly destroy an Innovative 
program. 

That which follows is an attempt to apply the notion of creative 
pessimism to the problems associated with the evaluation of Title One 
Programs in a large urban school district. In raising them, the author 
will be briefly serving as a resident skeptic. 

Although all of the problems examined may not be present in all 
large cities, they are by no means rare occurrences. Conversations 
with the individuals charged with the management of research and evaluation 
activities in large city school systems provides testament to this 
contention. 



Difficulties are arising elsewhere as well. In the September 197^ 
issue of Educational Researcher . John W. Evans of the U.S. Office of 
Education, wrote an article entitled "Evaluating Education Programs — 
Are We Getting Anywhere?" in which he points up some of the newer problems 
faced by educational evaluators. Though Evans was writing from the 
perspective of top-down. Federally sponsored research and evaluation 
activities, the concerns with which he deals more often than not, come 
home to roost in public school systems, for school systems are, after all, 
the sites of implementation of most such studies. Evans was particularly 
concerned with the following areas: 

1. As educational research and evaluation have proliferated, 
the people and institutions who are the objects of these 
studies have come under an increasing data collection burden - 
and are increasingly expressing their r^oistance to it. 

2. Evaluation studies that involve collecting data on adults are 
encountering increasing resistance at the interviewee level, 
particularly among minorities and the poor where it is now not 
uncommon for respondents to insist that they be paid for their 
time. 

3. The increased sensitivity to evaluation studies - both what 
they seek to find out and the amount of data they propose to 
collect - is resulting in a strangling growth of reviews, 
clearances, and advisory bodies. 

k. As protests over evaluations arise, osten-jibly over the type 
and amount of data to be collected, there is likely to be an 
increased pol iticization of these weapons in broader disputes. 
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5. Evaluation has the unfortunate effect on some program 
officers and school administrators of increasing their 
unwillingness to participate in such studies ror fear of 
what will happen to their programs if the evaluation produces 
negative findings. 

6. Evaluators are increasingly encountering unrealistic 
expectations on the part of policy-makers with respect to 
both the speed with which evaluations should be mounted and 
completed, and the simplicity of the answers which are desired. 

7. There is a great deal more public debate over the validity of 
evaluation methods and results. An increasingly important and 
time consuming task for evaluators will be defending the 
evaluations they carry out and their suitability as a basis 

for policy decisions. An unfortunate by-product of such debates 
is the impression created among both policymakers and the 
public that the mere fact such a debate is occurring means the 
evaluation must ipso facto be faulty and therefore should be 
put aside. 

Rather than merely present a lengthy list of anticipated concerns 
and .potential problems related to the evaluation of Title One Programs 
in a large urban school system, an attempt has been made to categorize 
each item under one of six rough headings — planning, organization, 
implementation, dissemination, personnel, and miscellaneous. The line 
between planning and implementations is often difficult to construct, 
especially since such 'concerns often must be dealt with during both 
phases of an evaluation. Nevertheless, the fallowing problems seemed to 



.oth Initially and most closely related to planning . 

, The failure of the Federal Government to be mor * » specific 
in mandating minimum evaluation requirements . This Is not all bad, 
for It allows the local educational agency to better tailor the 
evaluation to their particular information needs. However, it does 
create problems if the State agency serving as a conduit for Title 
One funds is given a free reign in establishing their interpretation 
of how the evaluation should be conducted 

.T he difficulty of producing useful data within a mandated timeframe . 
End-of-year evaluation reports are often required before funding 
for the following year is approved. When this is the case^ it becomes 
necessary to use data for the final report which were collected many 
months prior to the end of the school year, thus' creating an in- 
complete picture of the results. 

. The difficulty of nailing-down decision points and dates within the 
school system, so that data mi^ht be provided in time to impact 
upot decisions . Evaluation results may be required by half a dozen 
different decision-makers at different points in time. Without 
advance knowledge of this, the eva^uators cannot provide useful In- 
formation to their clients, and often end up producing inconclusive 
data instead. 

. Lack of support for the process of developing and implementing research 
and evaluation activities by instructional personnel . Because the 
office responsible for evaluation is usually separate from that charged 
with implementing the program, close coordination is essentia^ 
particularly in planning for the evaluation. Without it, the 
utility of the end product may be highly questionable. 



Closely related to problems associated with planning are those 
which impact upon system organization , 

, The dichotomy of responsibilities between high level decision-makers . 
The gray area between instructional planning and operations can 
result in key staff members utilizing only those evaluation results 
which will enlarge their personal domains of responsibility. 

. The lack of a formalized organizational linkage between the 
evaluation and instruction arm of the school system . This is 
particularly critical in terms of the feedback and utilization of 
mean i ngf ul i nf ormat i on . 

, Difficulties in accessing data collected in individual schools and 
by other divisions within the school system. Without a mandated 
process to control the types of data and ways in whrlch it Is collected, 
by divisions not charged with that responsibility, conflicting or 
Invalid results may be reported, evaluation over-kill may occur, 
and the reliability of all results may be jeopardized. 

, Tremendous numbers of requests for permission to conduct research 
V /ithin the school system . Government agencies, labs and centers, 
universities, and graduate students may account for over 500 
requests for access and/or cooperation each year. If even a 
majority of the better proposals were approved, the school system 
would quickly resemble an enormous fish tank, with school children 
playing the role of the fish. 

. Sharing data processing faci 1 i ties with the rest of the school system . 
When computer facilities are shared, business activities tend to 
receive first priority. Advance planning and scheduling may reduce 
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some of the deleterious effects of this upon the research 
and evaluation unit, but In emergencies, the payroll must 
always be run first, even though the funds used to pay for 
thousands of positions may depend upon the outcome of an 
evaluation. 

, The lack of zn office responsible for policy development . 
This leaves the school system without a formally constituted 
body to provide top level management with indications as to 
the complete ramifications which might be expected as a result 
of the implementation of new programs. A unit help key sta f 
translate evaluation findings into operational programs would 
enable middle and long range planning to be effecitv3ly carried out. 

It Is not surprising that the greatest number of concerns fall under 
the category of implementat ion . 

. Evaluation results not being considered at all in making 
program or management decisions . 

. The informal approach to management anyone ca n do it, and 
gut level Is the best way . This very often goes hand- in-hand 
with the above concern. The propensity of certain managers 
. to categorically Ignore Information from sources other than 
froiTi their own staffs Is perhaps the single most difficult 
problem with which evaluators must deal. A sub-set of this Is: 
. The inability of reticence of decision-makers to create new 
programs, or alter old ones, based upon evaluative data . 
In other words, ''don't confuse me with the facts.'* 
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. Evaluation data being ignored if It adversely aff ects 
patronage possibilities . Since many Title One Programs are 
personnel intensive, evaluation results which indicate that 
the va" in which certain personnel are utilized make little 
difference in meeting the program's objectives could result 
in the loss of jobs. If such jobs help establish a power base, 
there win lil^ely be tremendous resistance to doing away with 
them. 

. Resistance H;.t;. by supervisory personnel, includijijS. 
principals . Sometimes, such individuals may view the outcome 
of a program as totally their responsibility. Though identi- 
fying with the program is important, no Title One Program is 
evaluated on the basis of a single objective relating to the 
effectiveness of the program's support and leadership personnel. 
. The hands of decision-makers often bein g tied when they attempt 
to utilize evaluation results . This is sometimes due in part 
to the unwillingness of certain Board members to deal with 
results which might have political repercussions. 
. The voluntep-rino of the rp -«;e^rch unit by others to conduct 
surveys and provide data without first checking with them as 
to the feasibility of doing so and the resourc es available. 
Lack of control of resources, particularly time, can easily 
destroy the effectiveness of any organisation. School-based 
research and evaluation units are sometimes viewed as being 
similar to university researchers, and thus it is assumed 
that their staffs are sitting around, thinking, and waiting 
to be given something to do. In actuality, however. Title One 
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Program evaluations generally meet more rigid time tables than 
the programs themselves. 

, Criticism by city government that research and evaluation activities 
are an unnecessary part of the School District budget — a frill. 
In actuality the budget makes up only a fraction of ]% of the total 
school district's operating budget. All of the Title One Evaluation 
activities are mandated and paid for by federal categorical 
monies. 

The way in wich information is disseninated is a key element to 
any evaluation system. Because of its pivotal position, however, problems 
assoc?ated with dtsseminat ion often become even more ser:ous. 

. The philosopher stone syndrome the demand for gross over- 
simplification of results . The great majority of clients served 
by research and evaluation units, including the key decision-makers 
who use the data which they provide, would like researchers to 
tell them exactly what does and does not work. This is, of course, 
not possible without the specification of conditions and the 
identification of and controlling for the myriad variables 
associated with children and the environment in which they function. 
* , The demand that the evaluator provide an instructional prescription . 
This is closely related to the above concern. Though not an un- 
reasonable demand, instructional decisions are usually closely 
guarded by project personnel as being their sole responsibility. 
. The development of technical information in a form that will 
communicate to decision makers who are afraid of figures . 
Though evaluation reports must communicate in as simple and non* 
technical a form as possible it is not always feasible nor 
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advisable to completely eliminate all statistical information, 
, The media's propensity to over simplify and often alter information . 
In an effort to simplify often complicated results, the ''fourth 
estate^' quite regularly places emphasis upon less significant, 
though more sensational findings. Gross misrepresentation of data 
Is not uncommon as the result of the editor's scissors, 
, The expectation that "emergency information" will always bail 
out the requester . The "quicic, give me some data so I can quiet 
my critics" approach to evaluation Is just as likely to have the 
opposite effect. 

, The lack of understanding of the expectation and role of 
evaluation by federal funding sources , ESEA guidelines, for 
example, ere interpreted by many people as requiring pure research 
studies to be conducted on all funded projects, which Is, nor 
should it be, the case. 
Unless problems associated with personnel can be solved, the 
evaluation of Title One Programs cannot function properly. Examples 
of such concerns follow: 

," ^e fear of those in some way responsible for Title One Programs 
that an evaluation of said programs will discredit them. Evaluation 
results may be viewed as threatening by management personnel at 
a number of levels. Persons responsible for securing Federal 
doPars. cr.ay be concerned that they will be viewed as having "backed 
the wronq horse," Program directors, particularly those in charge 
of non-basic skills areas such as social studies, art, and music, 
may fear that their p ograms will be de-emphasized. Project managers 
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may associate a less than glowing project evaluation with certain 
discontinuation of funding. 

. Confusion between program evaluation and perso nnel ap^jusaj. 
This failure to differentiate between the evaluation of programs 
and the rating of people causes a tremendous amoun iculty 
In securing the cooperation vital to carrying out on-site evaluation 
activities. 

. The pervasive fear of the nn Mnn of accountability. Where systems 
of accountability are being implemented, they are not tied to 
evaluations geared toward other sets of objectives. Nevertheless, 
teachers' unions and administrators' associations have often 
erroneously related the function of a program evaluation to tha- 
of the office of personnel. 

. The threat of unionization of research and evalua tion personne l. 
into the same ba rn.-nino unit as teachers. This cou'd conceivably 
abrogate the ability of evaluators to observe and report findings, 
without being subject to undue pressures. 
ThP lack of trained educational res earch er s and evaluators. 
This creates staffing difficulties in all general program 
evaluation areas. The most problematic situation, however, is 
in identifying, recruiting, and hiring minority research and 
evaluation personnel. 

. Pressure to hire certain personnel and retain certain consultants. 
If th2 research and evaluation unit is used as a dumping ground, 
the quality of its produces wHl reflect the quality of the persons 
developing them. 
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Before concluding this series of potential problems, a number 
of miscel laneous concerns should also be raised. 

. Demands for non-existent information . Groups such as WRO and 
Parents* Union have on occasion demanded access to data which has 
never been collected. 

. Confidentiality and control of data . The impact of the Buckley 
amendment upon the ability of Title One research and evaluation 
units to cooperate in meaningful research projects may be 
cataclysmic. 

. Contracting with outside firm to provide evaluation services . 
Generally speal<ing, Title One Program evaluation services, if 
conducted by an autonomous in-house unit, can be provided better 
and more efficiently than contracting with an outside firm. 
What has preceded has been a deliberate attempt at pessimism. 
Hopefully, it hac been sufficiently creative to prove useful. However, 
the true test of its utility will rest with its ability to force those 
responsible for the creation and implementation of Title One Program 
evaluation services to anticipate problems and deal with them in advance, 
or at least develop strategies for doing so. 

It has not been the author's intention to demean the present "state 
of the art." To the contrary, many large cities are already dealing with 
the exigencies of Title One evaluation in spite of the difficulties they 
face. As the real benefits of evaluation become increasingly clear to the 
clients for whom such information is intended, many of the difficulties 
suggested in this paper will be dealt with by top management. Until that 
time, however, the role of the creative pessimist will be a vital one In 
facilitating the provision of information about Title One Programs. 
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